The main goal of this notebook is to use computer vision techniques in order to identify lanes in a variety of images and later on videos.
Here is one example of an image before and after being pre-processed.
|
|
This is the first asigment of the Self-Driving Car Engineer Nanodegree Program from Udacity
#importing some useful packages
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import os
import math
import numpy as np
import cv2
%matplotlib inline
Pretty simple function that resizes the images so all images have a standarized size for the pipeline of x=960 px by y=540 px. It is specially usefull when the boundaries of the regions of interest are defined by pixels.
Be aware that the origin of the images is in the upper right corner. Think of them as matrices not as images.
def resizeImage(image):
"""
image should be a np.array image, note that it will be modified
"""
ysize = image.shape[0]
xsize = image.shape[1]
# Resize image if necesaary
if xsize != 960 and ysize != 540:
image = cv2.resize(image, (960, 540),)
return image
Preview images to analize before start working on them
# Read list of images from a folder and obtain path
images_path ='test_images/'
names = os.listdir(images_path)
images_list = [images_path + item for item in names]
#Create matplotlib images from a list of paths
images = [mpimg.imread(item, cv2.IMREAD_COLOR) for item in images_list];
def displayListImages(images,titles,cols=1,rows=1,):
"""
Function to display and resize a list of images
images is a list of matplotlib image (use imread)
titles is a list of strings
cols is an integer with the number of columns to display
rows is an integer with the number of rows to display
"""
# Helper to adapt images to full width
plt.rcParams['figure.figsize'] = [12, 4*rows]
plt.rcParams['figure.dpi'] = 100 # 200 e.g. is really fine, but slower
for i in range(len(images)):
plt.subplot(rows, cols, i+1)
image = resizeImage(images[i])
plt.title(titles[i])
plt.imshow(image, cmap=None)
#Test function
displayListImages(images,cols=2,rows=6,titles=names)
Decent results can be obtained just in the RGB space but for a more robust approach, color enhancement is very effective.
A color enhancement is very usefull especially when looking for yellow lines, hence different techniques will be tested in order to find the best color filter to obtain the lines, specially under different tipes of tarmac or with the pressence of shades.
Isolate RGB color layers in the different color spaces: RGB, HSL, HSV to see wich ones contain onfo about line lanes.
The image below ilustrates de RGB, HSV and HSL color
|
|
|
In order to develop the color filters later on here the image below can be found with some explanation

*Important:*.
Be aware that the images to be able to be processed and filtered corrected should be 8 bit images
that is achieved using the following parameter when importing the image into a numpy array:
mpimg.imread(image_file_path, cv2.IMREAD_COLOR)
First of all we are going to detect wich color space has got more relevant features. For that we are going to use the the cell below how each color space transfroms the image and how much information is cointained in each color space plane by filter them induavidually.
imagesCEP = []
titlesCEP = []
# Use a tricky image to see wich color enhanment is best suited
imageTest = mpimg.imread(images_list[1], cv2.IMREAD_COLOR)
imageTest = resizeImage(imageTest)
#!!!!cv2.IMREAD_COLOR is crucial to be set so the image gets imported as 8 bits instead of floating point
# Change to different color spaces
hls = cv2.cvtColor(imageTest, cv2.COLOR_BGR2HLS)
hsv = cv2.cvtColor(imageTest, cv2.COLOR_BGR2HSV)
#rgb
imagesCEP.append(imageTest);titlesCEP.append("RGB");
imagesCEP.append(hls);titlesCEP.append("HLS");
imagesCEP.append(hsv);titlesCEP.append("HSV");
#rgb
imagesCEP.append(imageTest[:,:,0]);titlesCEP.append("imageR");
imagesCEP.append(imageTest[:,:,1]);titlesCEP.append("imageG");
imagesCEP.append(imageTest[:,:,2]);titlesCEP.append("imageB");
#hsl
imagesCEP.append(hls[:,:,0]);titlesCEP.append("hslH");
imagesCEP.append(hls[:,:,1]);titlesCEP.append("hslL");
imagesCEP.append(hls[:,:,2]);titlesCEP.append("hslS");
#hsv
imagesCEP.append(hsv[:,:,0]);titlesCEP.append("hsvH");
imagesCEP.append(hsv[:,:,1]);titlesCEP.append("hsvS");
imagesCEP.append(hsv[:,:,2]);titlesCEP.append("hsvV");
displayListImages(imagesCEP,titles=titlesCEP,cols=3,rows=4,)
Tip: If the features that we are interested in cannot be seen in one of the images below that means that that plane has got a lot of information about that feature and does not blend in with the rest of the image.
As it is possible to see in the iimage above the hsl color spaces seem to do a beter job finding yellow lines than the rbg space. Even the hsl and the hsv are prety similar, specially with the hue value where most of the information about the yellow line is contained, we are going to use the hsl as it becomes trivial to look for white lines.
Once the desired color space has been selected the next step is to develop a filter for the white pixels and another filter for yellow pixels then merge both with an or operation and then apply them to the original image with an and operation. Here is another good article on color filtering
In order to tune the filters, the l and s values are quite logical and work as expected in their range of 8 bits (0-255), but for the parameter h that should be in the range 0-180 (h=h/2) to make it fit in 8 bits I could not find a suitable map that correlates to the inputed values so it was basically trial and error.
# Cell to test color isolation
imageshsl = []
titleshsl = []
hls = cv2.cvtColor(imageTest, cv2.COLOR_BGR2HLS)
#find lleyow
# I think h is inverted as the H values shoudl be between 20 and 40 degrees for yellow and the range of the function is [0-180]
color1_hls = (80, 100, 0)
color2_hls = (100, 255, 255)
#Range HLS [0-180,0-254,0-254]
mask1 = cv2.inRange(hls, color1_hls,color2_hls)
imageshsl.append(mask1);titleshsl.append("mask Yellow");
#find white
color1_hls_w = (0, 200, 0)
color2_hls_w = (180, 255, 255)
#Range HLS [0-180,0-254,0-254]
mask2 = cv2.inRange(hls, color1_hls_w,color2_hls_w)
cv2.imwrite("mask2.png", mask2)
imageshsl.append(mask2);titleshsl.append("mask White");
# Add masks together
mask = cv2.bitwise_or(mask1, mask2)
#Apply mask to target
res = cv2.bitwise_and(imageTest,imageTest, mask= mask)
#plt.imshow(mask, cmap='gray') # this colormap will display in black / white
#plt.show()
#res = cv2.cvtColor(res, cv2.COLOR_BGR2GRAY)
imageshsl.append(res);titleshsl.append("res");
displayListImages(imageshsl,titles=titleshsl,cols=2,rows=2,)
As it can be seen the end result is pretty cool, in wich all the backgorund noise has been filtered out and basically just the line lanes and a couple of other features stand out.
Tip: Color enhancement is specially usefull when there are changes in the backgorund as tarmac varaitions or shadows as its able to filter out most of that noisy backgorund.
Note: Open cv uses BGR instead of RGB color channel by default.
def colorEnhancement(img):
"""Converts the image to HSL
Creates two masks to filter white and
lleyow lines
Applies the mask
Be carefull to input an 8bit image!
cv2.IMREAD_COLOR is your friend when using imgread
"""
hls = cv2.cvtColor(img, cv2.COLOR_BGR2HLS)
##find lleyows
color1_hls = (80, 100, 0)
color2_hls = (100, 255, 255)
mask1 = cv2.inRange(hls, color1_hls,color2_hls)
#find whites
color1_hls_w = (0, 200, 0)
color2_hls_w = (180, 255, 255)
mask2 = cv2.inRange(hls, color1_hls_w,color2_hls_w)
# Add masks together
mask = cv2.bitwise_or(mask1, mask2)
res = cv2.bitwise_and(img,img, mask= mask)
return res
#Test function
imagesCE=[]
titlesCE=[]
imageCE = colorEnhancement(imageTest)
imagesCE.append(imageTest);titlesCE.append("Original");
imagesCE.append(imageCE);titlesCE.append("Color enhanced");
displayListImages(imagesCE,cols=2,rows=1,titles=titlesCE)
As its name states, converts the 3 channel HSL or RGB (or even RGBA) to a single channel image. Even some information is lost on the way, it is very helpfull as it speeds up computation
def grayscale(img):
"""Applies the Grayscale transform
This will return an image with only one color channel
but NOTE: to see the returned image as grayscale
(assuming your grayscaled image is called 'gray')
you should call plt.imshow(gray, cmap='gray')"""
return cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
# Or use BGR2GRAY if you read an image with cv2.imread()
# return cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
#Test function
imagesGray=[]
titlesGray=[]
imageGray = grayscale(imageCE)
imagesGray.append(imageCE);titlesGray.append("Original");
imagesGray.append(imageGray);titlesGray.append("Gray scaled");
displayListImages(imagesGray,cols=2,rows=1,titles=titlesGray)
The end result is not very different (espceially when plotted in RGB), but computationally is going to be a good improvement.
def gaussian_blur(img, kernel_size):
"""Applies a Gaussian Noise kernel"""
return cv2.GaussianBlur(img, (kernel_size, kernel_size), 0)
#Test function
imagesGauss=[]
titlesGauss=[]
imageGauss = gaussian_blur(imageGray,5)
imagesGauss.append(imageGray);titlesGauss.append("Original");
imagesGauss.append(imageGauss);titlesGauss.append("Gaussian blurr. Kernel=5");
displayListImages(imagesGauss,cols=2,rows=1,titles=titlesGauss)
Lets see if we can make it easier for the computer to find the lines if we apply an edge detector. It is a very interesting detector that works with two thresholds.
It basically looks for gradient changes along the edges detected afted applying a 5x5 Gaussian filter.
After, it analices the full image by discarding the isloated gradient changes detected as edges,
Then uses two thresholds to make the edge neat, everything below the lower threshold is discarded. Everything above the high threshold is retaine. De data in between the lower and the upper is retained if continuous contours can be found. More info here

def cannyEdge(img, low_threshold=250,high_threshold=450):
imgCanny = cv2.Canny(img, low_threshold, high_threshold)
return imgCanny
#Test function
imagesCanny=[]
titlesCanny=[]
imageCanny = cannyEdge(imageGauss,250,450)
imagesCanny.append(imageGauss);titlesCanny.append("Original");
imagesCanny.append(imageCanny);titlesCanny.append("Canny edge. low th=250. high th=450");
displayListImages(imagesCanny,cols=2,rows=1,titles=titlesCanny)
Until now some filters have been applied but they are not enough to isolate just the lines from the rest of the picture, so some image masking for the area with the relevant features is required.
The Region of interes selected is a four side polygon wich can be shaped in width and height.
def region_of_interest(img, vertices):
"""
Applies an image mask.
Only keeps the region of the image defined by the polygon
formed from `vertices`. The rest of the image is set to black.
`vertices` should be a numpy array of integer points.
"""
#defining a blank mask to start with
mask = np.zeros_like(img)
#defining a 3 channel or 1 channel color to fill the mask with depending on the input image
if len(img.shape) > 2:
channel_count = img.shape[2] # i.e. 3 or 4 depending on your image
ignore_mask_color = (255,) * channel_count
else:
ignore_mask_color = 255
#filling pixels inside the polygon defined by "vertices" with the fill color
cv2.fillPoly(mask, vertices, ignore_mask_color)
#returning the image only where mask pixels are nonzero
masked_image = cv2.bitwise_and(img, mask)
return masked_image
#Test function
imagesROI=[]
titlesROI=[]
ysize =imageCanny.shape[0]
xsize =imageCanny.shape[1]
ROI_upperWidth = 100 #Width of the upper horizontal straight in px
ROI_upperHeight = 200 #Height of the upper horizontal straight from the bottom of the image in px
ROI_lowerWidth = 750 #Width of the lower horizontal straight in px
ROI_lowerHeight = 50 #Height of the lower horizontal straight from the bottom of the image in px
limitLL = ((xsize/2)-(ROI_lowerWidth/2),ysize-ROI_lowerHeight);
limitLR = (xsize - ((xsize/2)-(ROI_lowerWidth/2)),ysize-ROI_lowerHeight);
limitUL = ((xsize/2)-(ROI_upperWidth/2), ysize-ROI_upperHeight);
limitUR = ((xsize/2)+(ROI_upperWidth/2), ysize-ROI_upperHeight);
vertices = np.array([[limitLL,limitUL,limitUR , limitLR]], dtype=np.int32)
imageROI = region_of_interest(imageCanny,vertices)
imagesROI.append(imageCanny);titlesROI.append("Original");
imagesROI.append(imageROI);titlesROI.append("ROI");
displayListImages(imagesROI,cols=2,rows=1,titles=titlesROI)
# Plot vertices for verification in original image
x = [limitLL[0], limitUL[0], limitUR[0], limitLR[0],limitLL[0]]
y = [limitLL[1], limitUL[1], limitUR[1], limitLR[1],limitLL[1]]
plt.plot(x, y, 'b--', lw=4)
def weighted_img(img, initial_img, α=0.8, β=1., γ=0.):
"""
`img` is the output of the hough_lines(), An image with lines drawn on it.
Should be a blank image (all black) with lines drawn on it.
`initial_img` should be the image before any processing.
The result image is computed as follows:
initial_img * α + img * β + γ
NOTE: initial_img and img must be the same shape!
"""
return cv2.addWeighted(initial_img, α, img, β, γ)
def draw_lines(img, lines, color=[255, 0, 0], thickness=2):
"""
This function draws `lines` with `color` and `thickness`.
Lines are drawn on the image inplace (mutates the image).
If you want to make the lines semi-transparent, think about combining
this function with the weighted_img() function below
"""
for line in lines:
#print(line)
for x1,y1,x2,y2 in line:
cv2.line(img, (x1, y1), (x2, y2), color, thickness)
def hough_lines(img, rho, theta, threshold, min_line_len, max_line_gap):
"""
`img` should be a np.array image
Returns an image with hough lines drawn and the hough lines points.
"""
lines = cv2.HoughLinesP(img, rho, theta, threshold, np.array([]), minLineLength=min_line_len, maxLineGap=max_line_gap)
line_img = np.zeros((img.shape[0], img.shape[1], 3), dtype=np.uint8)
try:
draw_lines(line_img, lines, color=[255, 0, 0], thickness=4)
except:
lines = []
print("No line found")
pass
return line_img,lines
#Test function
imagesHough=[]
titlesHough=[]
# Hough parameters
rho = 1 # distance resolution in pixels of the Hough grid
theta = 1*np.pi/180 # angular resolution in radians of the Hough grid
threshold = 1 # minimum number of votes (intersections in Hough grid cell)
min_line_len = 5 #minimum number of pixels making up a line
max_line_gap = 40 # maximum gap in pixels between connectable line segments
imgHouhg,lines = hough_lines(imageROI, rho, theta, threshold, min_line_len, max_line_gap,)
imagesHough.append(imageROI);titlesHough.append("Original");
imagesHough.append(imgHouhg);titlesHough.append("Hough transform");
displayListImages(imagesHough,cols=2,rows=1,titles=titlesHough)
#print(lines)
Now that the points of the candidates for the line lanes are known, the next logical step is to draw those lines.
For that first a categorization is done based on the slope of the lines and the position in the screen. There is also a thresholdSlope to eliminate unwanted too shallow angled straights detected as lines.
Then a one dimensional fit algorithm is applied to extract the parameters of the best fitting straigh given by the categorized points.
Finally the equation of the straight y=mx+b is employed to extrapolate the points. As is more aesthetically visual that the lines have the same height the y points are fixed and the x points are calculated instead, inverting the equation: x=1/m -b/m.
Note: An axis transsformation is required when plotting the lines as the line parameters are calculated based on the origin being at the upper right corner.
def getSlope(line):
"""
Returns the slope of a line:
((y2-y1)/(x2-x1))
Usefull to decide which segments are part of the left
line vs. the right line.
"""
for x1,y1,x2,y2 in line:
slope = (y2-y1)/(x2-x1)
#print(x1,y1,x2,y2,slope)
return slope
# Adjust image output
plt.rcParams['figure.figsize'] = [12, 4]
# Categorize lines by slope
thresholdSlope = 0; #in degrees
thresholdSlope = math.tan(thresholdSlope*math.pi/180)
xR = [];yR = [];xL = [];yL = [];
limitLL, limitUL, limitUR, limitLR = vertices[0]
for line in lines:
x1,y1,x2,y2 = line[0];
currentSlope = getSlope(line);
if (currentSlope > thresholdSlope and x1 > limitUL[0] and x2 > limitUL[0]):
# Its a right line
if y1>y2:
x1,x2 = x2,x1
y1,y2 = y2,y1
xR += [x1, x2]
yR += [y1, y2]
elif (currentSlope < -1.0*thresholdSlope and x1 < limitUR[0] and x2 < limitUR[0] ):
# Its a left line
if y1<y2:
x1,x2 = x2,x1
y1,y2 = y2,y1
xL += [x1, x2]
yL += [y1, y2]
#print("xR:")
#print(xR)
#print("xL:")
#print(xL)
try:
zR = np.polyfit(xR, yR, 1)
mR, bR = zR
zR = 1/mR , -bR/mR
fR = np.poly1d(zR)
for x1, y1 in zip(xR, yR):
plt.plot(x1, y1, 'bo')
plt.plot((fR(limitUR[1]), fR(960)),(limitUR[1], 960), 'b')
except:
pass
try:
zL = np.polyfit(xL, yL, 1)
mL, bL = zL
zL = 1/mL , -bL/mL
fL = np.poly1d(zL)
for x1, y1 in zip(xL, yL):
plt.plot(x1, y1, 'ro')
plt.plot((fL(limitUL[1]), fL(960)),(limitUL[1], 960), 'r')
except:
pass
plt.axis([0, 960, 0, 540])
plt.yticks(rotation=0)
plt.xticks(rotation=0)
# Set origin in upper left
ax=plt.gca() # get the axis
ax.set_ylim(ax.get_ylim()[::-1]) # invert the axis
plt.show()
def linesFromCoeffs(z,side,vertices):
limitLL, limitUL, limitUR, limitLR = vertices[0]
f = np.poly1d(z)
if side == "right":
out = [int(f(limitUR[1])), int(limitUR[1]), int(f(960)),960]
else:
out = [int(f(limitUL[1])),int(limitUL[1]),int(f(960)), 960]
return out
def calculateLines(lines,vertices,thresholdSlope,zL_list=[],zR_list=[]):
"""
lines is the output of the hough_lines()
x1R is the x pixel where the right line ends/starts
x1L is the x pixel where the left line starts/ends
It splits the lines into right lane and left lane lines.
Then computes the best fit straight that fits all the input lane lines.
It returns the two lines to be drawn by its 4 points
If it cant compute the lines it returns two zero lenght lines.
"""
# Categorize lines by slope
thresholdSlope = math.tan(thresholdSlope*math.pi/180)
xR = [];yR = [];xL = [];yL = [];zL =[];zr=[];
global last_mL; global last_bL;
global last_mR; global last_bR;
limitLL, limitUL, limitUR, limitLR = vertices[0]
for line in lines:
x1,y1,x2,y2 = line[0];
currentSlope = getSlope(line);
if (currentSlope > thresholdSlope and x1 > limitUL[0] and x2 > limitUL[0]):
# Its a right line
if y1>y2:
x1,x2 = x2,x1
y1,y2 = y2,y1
xR += [x1, x2]
yR += [y1, y2]
elif (currentSlope < -1*thresholdSlope and x1 < limitUR[0] and x2 < limitUR[0]):
# Its a left line
if y1<y2:
x1,x2 = x2,x1
y1,y2 = y2,y1
xL += [x1, x2]
yL += [y1, y2]
# Parameters for video running mean
runningAvgCoef = 5
maxbDev = 50
maxmDev = 0.5
#Right line
try:
zR = np.polyfit(xR, yR, 1)
mR, bR = zR
zR = 1/mR , -bR/mR
except:
zR = last_mR , last_bR
pass
try:
if len(zR_list)<runningAvgCoef:
last_mR, last_bR = zR;
#Test for variations in b and m
if abs(abs(last_mR)-abs(zR[0])) < maxmDev and abs(abs(last_bR)-abs(zR[1])) < maxbDev:
zR_list.append(zR)
elif len(zR_list)>runningAvgCoef:
zR = last_mR , last_bR
else:
zR = 1/mR , -bR/mR
last_mR= zR[0]
last_bR= zR[1]
zR_list.append(zR)
# Apply moving average for videos
if len(zR_list)>runningAvgCoef:
zR_list_short = zR_list[-runningAvgCoef:]
last_mR = sum([item[0] for item in zR_list_short])/runningAvgCoef
last_bR = sum([item[1] for item in zR_list_short])/runningAvgCoef
zR = last_mR , last_bR
lRCoord = linesFromCoeffs(zR,'r',vertices)
except Exception as ex:
lRCoord = [0,0, 0, 0]
print(ex)
pass
#Left line
try:
zL = np.polyfit(xL, yL, 1)
mL, bL = zL
zL = 1/mL , -bL/mL
except:
zL = last_mL , last_bL
pass
try:
if len(zL_list)<runningAvgCoef:
last_mL, last_bL = zL;
#Test for variations in b and m
if abs(last_mL-zL[0]) < maxmDev and abs(last_bL-zL[1]) < maxbDev:
zL_list.append(zL)
elif len(zL_list)>runningAvgCoef:
zL = last_mL , last_bL
else:
zL = 1/mL , -bL/mL
last_mL= zL[0]
last_bL= zL[1]
zL_list.append(zL)
zL_list.append(zL)
# Apply moving average for videos
if len(zL_list)>runningAvgCoef:
zL_list_short = zL_list[-runningAvgCoef:]
last_mL = sum([item[0] for item in zL_list_short])/runningAvgCoef
last_bL = sum([item[1] for item in zL_list_short])/runningAvgCoef
zL = last_mL , last_bL
lLCoord = linesFromCoeffs(zL,'l',vertices)
except Exception as ex:
lLCoord = [0,0, 0, 0]
print(ex)
pass
return [[lLCoord,lRCoord]],zL,zR
#return [[[x1,y1, x2, y2],
# [x1,y1,x2,y2]]]
## Calculate output line lanes
## Test function
imagesLines=[]
titlesLines=[]
imgHouhg,lines = hough_lines(imageROI, rho, theta, threshold, min_line_len, max_line_gap)
if lines.any():
lines,zL,zR = calculateLines(lines,vertices,thresholdSlope=22)
#print(lines)
img_res_lines = np.copy(imgHouhg)*0
draw_lines(img_res_lines, lines, color=[0, 255, 0], thickness=8)
res = weighted_img(img_res_lines, imgHouhg, α=1, β=0.5, γ=0.)
imagesLines.append(imgHouhg);titlesLines.append("Original");
imagesLines.append(res);titlesLines.append("Lines drawn");
displayListImages(imagesLines,cols=2,rows=1,titles=titlesLines)
Once all the functions conforming the pipeline has been tested individually, its time to test it all toguether
# Read in the image and print out some stats
images_list = os.listdir("test_images/");
# Load images
image_path = 'test_images/' + images_list[1]
image_base = mpimg.imread(image_path, cv2.IMREAD_COLOR)
print('This image is: ',type(image_base),
'with dimensions:', image_base.shape)
###################### Pipeline start #################
pipImages = []
piptitles = []
###### Resize image
image_base = resizeImage(image_base)
imagesCE.append(image_base);titlesCE.append("Original");
###### Color Enhancement
imageCE = colorEnhancement(image_base)
pipImages.append(imageCE);piptitles.append("Color Enhanced");
###### GrayScale
imageGray = grayscale(imageCE)
pipImages.append(imageGray);piptitles.append("Image Gray");
###### Gauss Smoothing
imageGauss = gaussian_blur(imageGray,5)
pipImages.append(imageGauss);piptitles.append("Image Gauss");
###### Canny Edge
imageCanny = cannyEdge(imageGauss,250,450)
pipImages.append(imageCanny);piptitles.append("Canny Edge");
###### ROI
ysize =imageCanny.shape[0]
xsize =imageCanny.shape[1]
ROI_upperWidth = 100 #Width of the upper horizontal straight in px
ROI_upperHeight = 200 #Height of the upper horizontal straight from the bottom of the image in px
ROI_lowerWidth = 750 #Width of the lower horizontal straight in px
ROI_lowerHeight = 50 #Height of the lower horizontal straight from the bottom of the image in px
limitLL = ((xsize/2)-(ROI_lowerWidth/2),ysize-ROI_lowerHeight);
limitLR = (xsize - ((xsize/2)-(ROI_lowerWidth/2)),ysize-ROI_lowerHeight);
limitUL = ((xsize/2)-(ROI_upperWidth/2), ysize-ROI_upperHeight);
limitUR = ((xsize/2)+(ROI_upperWidth/2), ysize-ROI_upperHeight);
vertices = np.array([[limitLL,limitUL,limitUR , limitLR]], dtype=np.int32)
imageROI = region_of_interest(imageCanny,vertices)
pipImages.append(imageROI);piptitles.append("ROI");
###### Hough lines
rho = 1 # distance resolution in pixels of the Hough grid
theta = 1*np.pi/180 # angular resolution in radians of the Hough grid
threshold = 15 # minimum number of votes (intersections in Hough grid cell)
min_line_len = 5 #minimum number of pixels making up a line
max_line_gap = 40 # maximum gap in pixels between connectable line segments
imgHouhg,lines = hough_lines(imageROI, rho, theta, threshold, min_line_len, max_line_gap)
pipImages.append(imgHouhg);piptitles.append("Houhg lines");
###### Lane lines
if lines.any():
linesArrow,zL,zR = calculateLines(lines,vertices,thresholdSlope=22)
#print(lines)
img_res_lines = np.copy(imgHouhg)*0
draw_lines(img_res_lines, linesArrow, color=[0, 255, 0], thickness=8)
res = weighted_img(img_res_lines, image_base, α=1, β=0.5, γ=0.)
pipImages.append(res);piptitles.append("Result");
displayListImages(pipImages,cols=2,rows=4,titles=piptitles)
That is it basically, once the pipeline has been correctly tuned, it can be applied to the original images to see the ressults
def process_image(img,canny_low=50,canny_high=150,
ROI_upperWidth=120,
ROI_upperHeight=210,
ROI_lowerWidth=750,
ROI_lowerHeight=50,
GSKernel =7,
hough_rho = 1,
hough_theta = 1,
hough_threshodl = 10,
hough_min_line_len = 5,
hough_max_line_gap = 10,
thresholdSlope=20,
plots = False,
video= True
):
#### Frame dependence for video
if not video:
global zL_list
zL_list= []
global zR_list
zR_list= []
###################### Pipeline start #################
###### Resize image
img = resizeImage(img)
###### Color Enhancement
imageCE = colorEnhancement(img)
###### GrayScale
imageGray = grayscale(imageCE)
###### Gauss Smoothing
imageGauss = gaussian_blur(imageGray,GSKernel)
###### Canny Edge
imageCanny = cannyEdge(imageGauss,canny_low,canny_high)
###### ROI
ysize =imageCanny.shape[0]
xsize =imageCanny.shape[1]
limitLL = ((xsize/2)-(ROI_lowerWidth/2),ysize-ROI_lowerHeight);
limitLR = (xsize - ((xsize/2)-(ROI_lowerWidth/2)),ysize-ROI_lowerHeight);
limitUL = ((xsize/2)-(ROI_upperWidth/2), ysize-ROI_upperHeight);
limitUR = ((xsize/2)+(ROI_upperWidth/2), ysize-ROI_upperHeight);
vertices = np.array([[limitLL,limitUL,limitUR , limitLR]], dtype=np.int32)
imageROI= region_of_interest(imageCanny, vertices)
# Plot vertices for verification in original image
x = [limitLL[0], limitUL[0], limitUR[0], limitLR[0],limitLL[0]]
y = [limitLL[1], limitUL[1], limitUR[1], limitLR[1],limitLL[1]]
###### Hough lines
theta = hough_theta*np.pi/180 # angular resolution in radians of the Hough grid
imgHouhg,lines = hough_lines(imageROI, hough_rho, hough_theta, hough_threshodl, hough_min_line_len, hough_max_line_gap,)
imgHouhg = weighted_img(imgHouhg, img, α=1, β=0.9, γ=0.)
###### Lane lines
if len(lines)>0:
linesArrow,zL,zR = calculateLines(lines,vertices,thresholdSlope,zL_list,zR_list)
#print(lines)
res = np.copy(img)*0
draw_lines(res, linesArrow, color=[0, 255, 0], thickness=8)
#imgHouhg = weighted_img(res, imgHouhg, α=1, β=0.5, γ=0.)
image_out = weighted_img(res, img, α=1, β=0.5, γ=0.)
else:
image_out = img
###### If plots are required
if plots:
# Helper to adapt images to full width
plt.rcParams['figure.figsize'] = [12, 8]
plt.rcParams['figure.dpi'] = 100 # 200 e.g. is really fine, but slower
# Helper function to display images correctly
fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2)
ax1.imshow(imageCE, cmap=None)
ax2.imshow(imageCanny, cmap=None)
ax1.plot(x, y, 'b--', lw=4)
ax3.imshow(imgHouhg)
ax4.imshow(image_out)
return
return image_out
#Test function
a = process_image(images[0],plots=False, video=False)
plt.figure()
plt.imshow(a, cmap=None)
# or
process_image(images[0],plots=True,video=False)
Interactive function to tune parameters to optimize line detection.
from ipywidgets import *
from IPython.display import display
# Read in the image and print out some stats
images_list = os.listdir("test_images/");
# Load images
image_path = 'test_images/' + images_list[1]
image_root = mpimg.imread(image_path, cv2.IMREAD_COLOR)
#Define sliders
canny_low_slider = IntSlider(value=250,min=0,max=500,step=1,continuous_update=False,)
canny_high_slider = IntSlider(value=450,min=0,max=1000,step=1,continuous_update=False,)
GSKernel_slider = IntSlider(value=5,min=1,max=55,step=2,continuous_update=False,)
hough_rho_slider = FloatSlider(value=1.0,min=0.1,max=10,step=0.1,continuous_update=False,)
hough_theta_slider = FloatSlider(value=1.0,min=0.1,max=10,step=0.1,continuous_update=False,)
hough_threshodl_slider = IntSlider(value=15,min=0,max=100,step=1,continuous_update=False,)
hough_min_line_len_slider = IntSlider(value=5,min=0,max=400,step=1,continuous_update=False,)
hough_max_line_gap_slider = IntSlider(value=40,min=0,max=400,step=1,continuous_update=False,)
thresholdSlope_slider = IntSlider(value=0,min=0,max=90,step=1,continuous_update=False,)
#Create interactive function
def interactive(canny_low,
canny_high,
GSKernel,
hough_rho,
hough_theta,
hough_threshodl,
hough_min_line_len,
hough_max_line_gap,
thresholdSlope):
return process_image(image_root,
canny_low=canny_low,
canny_high=canny_high,
GSKernel=GSKernel,
hough_rho=hough_rho,
hough_theta=hough_theta,
hough_threshodl=hough_threshodl,
hough_min_line_len=hough_min_line_len,
hough_max_line_gap=hough_max_line_gap,
thresholdSlope=thresholdSlope,
plots = True,
video=False,)
# Run interacctive function
interact(interactive, canny_low=canny_low_slider,
canny_high=canny_high_slider,
GSKernel=GSKernel_slider,
hough_rho=hough_rho_slider,
hough_theta=hough_theta_slider,
hough_threshodl=hough_threshodl_slider,
hough_min_line_len=hough_min_line_len_slider,
hough_max_line_gap=hough_max_line_gap_slider,
thresholdSlope = thresholdSlope_slider,
)
Results applied to the initial images
# Read list of images from a folder and obtain path
images_path ='test_images/'
names = os.listdir(images_path)
images_list = [images_path + item for item in names]
#Create matplotlib images from a list of paths
images = [mpimg.imread(item, cv2.IMREAD_COLOR) for item in images_list];
results = []
for i in range(len(images)):
res = process_image(images[i], plots = False,video=False)
results.append(res)
mpimg.imsave('test_images_output/' + names[i],res)
displayListImages(results,cols=2,rows=6,titles=names)
The results are quite good it generally tracks the lines correctly (if they are vissible). There is a bit of wobling around dashed lines as dots can be ignored sometimes derivating in a slight change in angle.
Overall the pipeline works as expected detecting white and yellow line markigs and sucessfully filtering out most of the background noise, even with shadows and tarmac changes
# Import everything needed to edit/save/watch video clips
from moviepy.editor import VideoFileClip
from IPython.display import HTML
# Necessary to reset previous frame properties to carry out moving average smoothing
global zR_list
zR_list = []
global zL_list
zL_list = []
white_output = 'test_videos_output/solidWhiteRight.mp4'
## To speed up the testing process you may want to try your pipeline on a shorter subclip of the video
## To do so add .subclip(start_second,end_second) to the end of the line below
## Where start_second and end_second are integer values representing the start and end of the subclip
## You may also uncomment the following line for a subclip of the first 5 seconds
#clip1 = VideoFileClip("test_videos/solidWhiteRight.mp4").subclip(0,2)
clip1 = VideoFileClip("test_videos/solidWhiteRight.mp4")
white_clip = clip1.fl_image(process_image) #NOTE: this function expects color images!!
%time white_clip.write_videofile(white_output, audio=False)
Play the video inline, or if you prefer find the video in your filesystem (should be in the same directory) and play it in your video player of choice.
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format(white_output))
Now for the one with the solid yellow lane on the left. This one's more tricky!
# Necessary to reset previous frame properties
global zR_list
zR_list = []
global zL_list
zL_list = []
yellow_output = 'test_videos_output/solidYellowLeft.mp4'
## To speed up the testing process you may want to try your pipeline on a shorter subclip of the video
## To do so add .subclip(start_second,end_second) to the end of the line below
## Where start_second and end_second are integer values representing the start and end of the subclip
## You may also uncomment the following line for a subclip of the first 5 seconds
#clip2 = VideoFileClip('test_videos/solidYellowLeft.mp4').subclip(0,5)
clip2 = VideoFileClip('test_videos/solidYellowLeft.mp4')
yellow_clip = clip2.fl_image(process_image)
%time yellow_clip.write_videofile(yellow_output, audio=False)
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format(yellow_output))
Try your lane finding pipeline on the video below. Does it still work? Can you figure out a way to make it more robust? If you're up for the challenge, modify your pipeline so it works with this video and submit it along with the rest of your project!
# Necessary to reset previous frame properties
global zR_list
zR_list = []
global zL_list
zL_list = []
challenge_output = 'test_videos_output/challenge.mp4'
## To speed up the testing process you may want to try your pipeline on a shorter subclip of the video
## To do so add .subclip(start_second,end_second) to the end of the line below
## Where start_second and end_second are integer values representing the start and end of the subclip
## You may also uncomment the following line for a subclip of the first 5 seconds
##clip3 = VideoFileClip('test_videos/challenge.mp4').subclip(0,2)
clip3 = VideoFileClip('test_videos/challenge.mp4')
challenge_clip = clip3.fl_image(process_image)
%time challenge_clip.write_videofile(challenge_output, audio=False)
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format(challenge_output))
clip3.save_frame("test_videos/challenge_7s.png", t=7) # saves the frame a t=xs
yellow_clip.write_gif("test_videos_output/solidYellowLeft.gif",fps=15)
white_clip.write_gif("test_videos_output/solidWhiteRight.gif",fps=15)
challenge_clip.write_gif("test_videos_output/challenge_.gif",fps=15)
cv2.inRange() for color selection
cv2.fillPoly() for regions selection
cv2.line() to draw lines on an image given endpoints
cv2.addWeighted() to coadd / overlay two images
cv2.cvtColor() to grayscale or change color
cv2.imwrite() to output images to file
cv2.bitwise_and() to apply a mask to an image